Comments about the article in Nature: How Covid Broke the Evidence Pipeline

Following is a discussion about this article in Nature Vol 593 13 May 2021, by Helen Pearson
To study the full text select this link: https://www.nature.com/articles/d41586-021-01246-x In the last paragraph I explain my own opinion.

Contents

Reflection

Introduction

Carley is also a specialist in evidence-based medicine — the transformative idea that physicians should decide how to treat people by referring to rigorous evidence, such as clinical trials.
Clinical trials are the standard procedure to test new drugs. https://en.wikipedia.org/wiki/Clinical_trial#Phases
As cases of COVID-19 climbed in February, Carley thought that clinicians were suddenly abandoning evidence and reaching for drugs just because they sounded biologically plausible.
Interesting information. Why?
Early studies Carley saw being published often lacked control groups or enrolled too few people to draw firm conclusions.
Part of the problem was haste, because of the serious and dangerous disease involved.
Organizations worldwide have scrambled to synthesize the available evidence on drugs, masks and other key issues, but can’t keep up with the outpouring of new research, and often repeat others’ work.
I can understand that a certain coordination plan is required.
At the same time, shining examples of good practice have emerged: medical advances based on rigorous evidence have helped to chart a route out of the pandemic.
What is the definition of rigorous evidence in thse cases?
The rapid trials of vaccines were spectacular successes, and well-run trials of possible treatments have shown, for instance, that some steroids help to fight COVID-19, but the drug hydroxychloroquine doesn’t.
What is the evidence that steroids help and hydrochloroquine doesn't ?
COVID-19 is a stress test that revealed the flaws in systems that produce evidence, says Elie Akl, an internal-medicine specialist and clinical epidemiologist at the American University of Beirut. “It would be shameful if we come out of this experience and not make the necessary change for the next crisis.”
Now we need more detail.

1. The evidence revolution

Many doctors practising today weren’t taught too much about clinical trials in medical school. It was standard to offer advice largely on the basis of opinion and experience, which, in practice, often meant following the advice of the most senior physician in the room. (Today, this is sometimes called eminence-based medicine.)
Okay. In principle there is nothing wrong with this. In priciple the advice should not come from 'the room' but from 'the whole world'
But he found out later that what he’d been taught was wrong: six controlled clinical trials had shown that antibiotics given early to children with measles were effective at preventing serious bacterial infections.
What you need is one study group which uses the old method and one study group which uses the new method. The study group which has the best results should be identified as the prefered method. If this is the new method, than this is the one that should be more thourougly investigated how it works.
It was a landmark study, and in 1993, Chalmers was central in founding the Cochrane collaboration, which aimed to follow this model and synthesize evidence across other medical specialties.
Bravo.
Organizations in areas ranging from education to conservation also create evidence syntheses, and policymakers find them an invaluable tool.
Okay.
When faced with a slew of conflicting studies, an evidence synthesis “has the power to identify important conclusions about what works that would never be possible from assessing the underlying trials in isolation”, says Karla Soares-Weiser, editor-in-chief of the Cochrane Library and acting chief executive of Cochrane, based in Tel Aviv, Israel.
When there are conflicting studies a much more, something like an umbrella study, should be set up globally.

2. Tumult of trials

Carley compares the time before and after COVID-19 to a choice of meals.
Okay.
“They didn’t know where it came from or what the ingredients were, they weren’t entirely sure whether it was meat or vegetarian, they would just eat anything,” he says.
This sounds like there was chaos. That is strange, because COVID-19, is similar as many other COVID virussen. The main problem is that initinally people had the disease, without knowing this and by using airplanes were spreading the disease at an enormous speed, all over the world.
It was “hype-based medicine”, he says — fuelled by former US president Donald Trump, who announced last May that he had started taking the drug himself.
That is a problem of the doctors themself. They should use common standard practice.
Many doctors and researchers did race to launch clinical trials — but most were too small to produce statistically meaningful results, says Tikkinen, who leads the Finnish arm of SOLIDARITY, an international clinical trial of COVID-19 treatments coordinated by the World Health Organization (WHO).
I think there was a different problem. Many people tried testing existing drugs, to see if they helped against COVID-19 which is a deadly disease. The more people you test in two groups, at least a certain number of people will die in the group who don't get this 'new' drug.
Hydroxychloroquine was the most-tested drug according to a database of 2,900 COVID-19 clinical trials called COVID-NMA: it was tested in 250 studies involving nearly 89,000 people (see ‘Too many trials?’). Many are still under way, despite convincing evidence that the drug doesn’t help: the RECOVERY trial concluded that hydroxychloroquine should not be recommended to treat COVID-19 in June last year.
If the average dead rate in that period was 20 per 1000 people, than as a cause of these studies roughly 2000 people died of COVID-19, because hydroxychloroquine was no cure.
Instead, hospitals should have joined up, as was done in a handful of mega-trials... in part because it was kept simple — a short consent procedure and one outcome measure: death within 28 days of being randomly assigned to a treatment or control group. The trial has now enrolled nearly 40,000 people at 180 sites and its results showing that the steroid dexamethasone reduced death rates changed standard practice almost overnight.
In this case, without the cure, assuming 20 deads per 1000 people, 800 people should have died. With the cure still 400 people should have died.
One clear take-home lesson, researchers say, is that countries need more large-scale national and international clinical-trial protocols sitting on the shelf, ready to fire up quickly when a pandemic strikes. “We will learn a lot of lessons from this, and I think RECOVERY has set the standard,” Tikkinen says.
When this is only lesson and when theres is a new pandemic (of a new virus) I don't think this is the way to go.

3. The rise of reviews

The pandemic is “evidence on steroids”, says Gabriel Rada, who directs the evidence-based health-care programme at the Pontificial Catholic University of Chile in Santiago. Research on the disease has been produced at a phenomenal rate. And that created a knock-on problem for researchers who try to make sense of it.
I can imagine.
Rada runs a giant database of systematic reviews in health called Epistemonikos (a Greek term meaning ‘what is worth knowing’). It now contains nearly 9,000 systematic reviews and other evidence syntheses related to COVID-19.
Interesting, but a cause of problems
He counted more than 100 on hydroxychloroquine, all out of date. “You have this huge amount of inappropriate and probably wasteful duplication of effort,” Grimshaw says. “There’s a fundamental noise-to-signal problem.”
I agree.
Researchers already knew that evidence syntheses took too long to produce and fell quickly out of date, and the pandemic threw those problems into sharp relief. Cochrane’s median time to produce a review is more than two years and, although it commits to updating them, that isn’t nimble enough when new research is flooding out. So, during the pandemic, Cochrane cut the time of some reviews to three to six months.
Okay.
Drawing on this and other sources, a handful of groups including Cochrane have been developing living systematic reviews. Siemieniuk had produced such reviews before and helped to convene a group to build one on COVID-19 therapies. The international team, now about 50–60 people, combs the literature daily for clinical trials that could change practice and distils findings into a living guideline that doctors can quickly refer to at a patient’s bedside and which is used by the WHO.
The issue is still clinical studies. The major problem is that when there are many studies done, how do you compare each with all the other ones. To solve that (?) when someone issues a document he or she should identify which one, of the old studies, this new study resembles the most.
Suppose in one study the number of deads in 1000 patients, untreated versus treated is 20 versus 0 and in a different study also 20 versus 0, which one do you select? The cause is the number of patients is too small to make a clear selection. The solution is to test more people. The problem is when you test 10000 patients you still get 200 deads, so you can ask your self the question if this is the right way to go.
...Grimshaw, who co-leads COVID-END, argues that ... “I think there’s a strong argument that you’ll get more bang for the buck if, in selected areas, you invest in living reviews,” he says.
Maybe.

4. Mosaic of evidence

In the United Kingdom, Trish Greenhalgh, a health researcher and doctor at the University of Oxford, expressed frustration at those who wanted bullet-proof evidence from randomized controlled trials before recommending the widespread use of face masks, even though there was a wealth of other evidence that masks could be effective and, unlike an experimental drug, that they posed little potential harm.
I can understand the frustration of Trish Greenhalgh that people need bullet-proof evidence that the use of face masks have a positif influence in the strugle against Covid-19. The most probable way that the corona virus is transmitted between humans is directly, by means of airdroplets which contain the virus. The cycle starts because someone is infected. The best way to stop the cycle is to wear a face mask, by the infected person, to protect the nose and mouth from spreading the virus. A more practical solution is that everyone should wear a face mask.
There are many ways to prevent the spreading of COVID-19, including face masks, travelling, distancing and handwashing. To actual demonstrate by means of an experiment, which ones are the best is difficult.
“I think that was just a blast of common sense,” says David Tovey, co-editor in chief of the Journal of Clinical Epidemiology and an adviser to COVID-END, based in London.
It is wrong to use the term common sense. When there are different medicines to treat a disease one way to go is to test the treatment to decide which one is the best. A different approach is to study how does each of the medicines function.
When you want to study how a virus performs his deadly task one way to study is how does the virus multiply in the human body. We know that the human body is capable by attacking this path way, defending him self, via antibodies. etc etc.
“People have focused too much on randomized trials as being the one source of truth.”
See: Reflection 1 - Evidence based science (medicin) versus facts based science
See: Reflection 2 - Facts based science in relation to COVID 19.
But in many situations, such trials are unethical, impractical or unfeasible: it’s impossible to do a randomized controlled trial to test whether a new urban motorway improves people’s health by siphoning traffic out of town, for example.
Exactly. It seems to me that people forgot how to prevent infectious diseases.
Often, researchers have to pragmatically assess a range of different evidence — surveys, natural experiments, observational studies and trials — and mosaic them together to give a picture of whether something is worthwhile. “You have to get on and do what we can with the best available evidence, then continue to evaluate what we’re doing,” says Ogilvie.
The problem is that COVID-19 is a deadly disease, as already is mentioned, and that it is very difficult to test how you can prevent its spreading. It is much more important to investigate how its spreads outside the human body, between humans, and secondly within the body.
Between humans it spreads directly via the mouth and indirectly via contaminated surfaces.
The primary way to prevent speading is via face masks and secondly via distancing and minizing travelling. In fact the best option is no travelling at all.

To quantify these options is difficult, if not impossible, specific if more measures are involved.

“We’re definitely fighting against big forces,” says Per Olav Vandvik, who heads the MAGIC Evidence Ecosystem Foundation in Oslo, which supports the use of trustworthy evidence.
It is better 'to support the use of quality facts'
Leaders in the field will pick up these debates in October during the virtual meeting organized by Cochrane, COVID-END and the WHO, to discuss what has been learnt about evidence supply and demand during the pandemic — and where to go next.
It is better to discuss: "What did we learn about the pandemic. What did we wrong and what did we good." With we I mean a certain group of people every where on planet earth. For example:

Reflection 1 - Evidence based science (medicin) versus facts based science

In science the two concepts 'to prove something' and 'what is the evidence' are difficult in its practical usage. In criminal science the police inspector needs evidence who commited the murder. Is that really what he wants? IMO what he wants is what has happened that causes Mr X to die. What he wants are the true facts.
In physics you can ask a similar question: Is it possible to prove that water boils at 100 degrees. The answer is No. The only thing that we can do is to demonstrate that when you heat water its starts to boil and the amount of water decreases and when you cool water that it becomes ice. This are facts.
This are two example which identify how important the facts are. In general you get the facts by observing what has happened by performing experiments and by describing the experiments as detailed as possible.

A very important set of facts is to describe what happens when (almost) nothing is done. Such a study is important to compare the results when something is done. Such a study is also important to observe how a healthy comunity evolves.

Reflection 2 - Facts based science in relation to COVID 19.

Facts based science in relation COVID 19 involves four areas:
  1. How does the disease spread from human to human.
  2. How does the disease spreads within the body.
  3. Development and testing of a medicin against Covid 19.
  4. Development and testing of a vaccin.
    • 4a. Is the vaccin safe for the person who is vaccinated.
    • 4b. Does the vaccin prevent spreading of Covid 19.
Of all these areas #4b is the most difficult, because it requires that you bring vaccinated people in contact with non-vaccinated people. In principle with people who can have Covid 19 and what is the most important, no extra protection measures should be enforced. That means no masks should be used and no distance measures.

The question is how do you set-up a test that vaccinated people are less prone to get Covid 19?
You start with 1000 people and 500 people get a vaccin and 500 get a placebo. The problem is that the vaccination does not work instantaneous that means all the vaccinated people should go in quarantaine. However you must do the same with the non-vaccinated people. You should treat everyone the same. But then most probably after that week no one has Covid 19 any more, because all the ones that get COVID 19 are 'removed'. When you bring thereafter, all the 1000 people together (minus the removed people), there is a high chance that no one gets the disease. However, now you have a problem, because this does not prove that the vaccinated actual worked. This shows how difficult working with two groups is.

Situation 4a is different and simpler, because if vaccinated people get any side effects and the non-vaccinated not than the cause is the vaccin. In order to detect side effects (specific if different age groups are considerd) than a group of 1000 people is too small. If people have to go in quarantaine is no issue. This is also the case when face masks have to be used or to keep distance. All of this makes situation #4 much simpler than #4b.

If you want to give a comment you can use the following form Comment form

Created: 30 March 2021

Back to my home page Index
Back to Nature comments Nature Index